{
 "metadata": {
  "name": "convolutional_network"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# pylearn2 tutorial: Convolutional network\n",
      "by [Ian Goodfellow](http://www-etud.iro.umontreal.ca/~goodfeli)\n",
      "\n",
      "## Introduction\n",
      "This ipython notebook will teach you the basics of how convolutional networks work, and show you how to use multilayer perceptrons in pylearn2.\n",
      "\n",
      "To do this, we will go over several concepts:\n",
      "\n",
      "Part 1: What pylearn2 is doing for you in this example\n",
      "\n",
      "   - Review of multilayer perceptrons, and how convolutional networks are similar\n",
      "\n",
      "   - Convolution and the equivariance property\n",
      "\n",
      "   - Pooling and the invariance property\n",
      "\n",
      "   - A note on using convolution in research papers\n",
      "\n",
      "Part 2: How to use pylearn2 to train a convolutional network\n",
      "\n",
      "    - pylearn2 Spaces\n",
      "\n",
      "    - MNIST classification example\n",
      "\n",
      "\n",
      "Note that this won't explain in detail how the individual classes are implemented. The classes\n",
      "follow pretty good naming conventions and have pretty good docstrings, but if you have trouble\n",
      "understanding them, write to me and I might add a part 3 explaining how some of the parts work\n",
      "under the hood.\n",
      "\n",
      "Please write to pylearn-dev@googlegroups.com if you encounter any problem with this tutorial."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Requirements\n",
      "\n",
      "Before running this notebook, you must have installed pylearn2.\n",
      "Follow the [download and installation instructions](http://deeplearning.net/software/pylearn2/#download-and-installation)\n",
      "if you have not yet done so.\n",
      "\n",
      "This tutorial also assumes you already know about multilayer perceptrons, and know how to train and evaluate a multilayer perceptron in pylearn2. If not, work through multilayer_perceptron.ipynb before starting this tutorial.\n",
      "\n",
      "It's also strongly recommend that you run this notebook with THEANO_FLAGS=\"device=gpu\". This is a processing intensive example and the GPU will make it run a lot faster, if you have one available. Execute the next cell to verify that you are using the GPU.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import theano\n",
      "print theano.config.device"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "gpu\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "Using gpu device 0: GeForce GTX 285\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Part 1: What pylearn2 is doing for you in this example\n",
      "\n",
      "In this part, we won't get into any specifics of pylearn2 yet. We'll just discuss what a convolutional network is. If you already know about convolutional networks, feel free to skip to part 2.\n",
      "\n",
      "\n",
      "### Review of multilayer perceptrons, and how convolutional networks are similar\n",
      "\n",
      "In multilayer_perceptron.ipynb, we saw how the multilayer perceptron (MLP) is a versatile model that can do many things. In this series of tutorials, we think of it as a classification model that learns to map an input vector $x$ to a probability distribution $p(y\\mid x)$ where $y$ is a categorical value with $k$ different values. Using a dataset $\\mathcal{D}$ of $(x, y)$, we can train any such probabilistic model by maximizing the log likelihood,\n",
      "\n",
      "$$ \\sum_{x,y \\in \\mathcal{D} } \\log P(y \\mid x). $$\n",
      "\n",
      "The multilayer perceptron defines $P(y \\mid x)$ to be the composition of several simpler functions. Each function being composed can be thought of as another \"layer\" or \"stage\" of processing.\n",
      "\n",
      "A convolutional network is nothing but a multilayer perceptron where some layers take a very special form, which we will call \"convolutional layers\". These layers are specially designed for processing inputs where the indices of the elements have some topological significance.\n",
      "\n",
      "For example, if we represent a grayscale image as an array $I$ with the array indices corresponding to physical locations in the image, then we know that the element $I_{i,j}$ represents something that is spatially close to the element $I_{i+1,j}$. This is in contrast to a vector representation of an image. If $I$ is a vector, then $I_i$ might not be very close at all to $I_{i+1}$, depending on whether the image was converted to vector form in row-major or column major format and depending on whether $i$ is close to the end of a row or column.\n",
      "\n",
      "Other kinds of data with topological in the indices include time series data, where some series $S$ can be indexed by a time variable $t$. We know that $S_t$ and $S_{t+1}$ come from close together in time. We can also think of the (row, column, time) indices of video data as providing topological information.\n",
      "\n",
      "Suppose $T$ is a function that can translate (move) an input in the space defined by its indices by some amount $x$.\n",
      "In other words,\n",
      "$T(S,x)_i = S_j$ where $j=i-x$ (a MathJax or ipython bug seems to prevent me from putting $i-x$ in a subscript).\n",
      "Convolutional layers are an example of a function $f$ designed with the property $f(T(S,x)) \\approx f(S)$ for small x.\n",
      "\n",
      "This means if a neural network can recognize a handwritten digit in one position, it can recognize it when it is slightly shifted to a nearby position. Being able to recognize shifted versions of previously seen inputs greatly improves the generalization performance of convolutional networks.\n",
      "\n",
      "\n",
      "## Convolution and the equivariance property\n",
      "\n",
      "TODO\n",
      "\n",
      "## Pooling and the invariance property\n",
      "\n",
      "TODO\n",
      "\n",
      "## A note on using convolution in research papers\n",
      "\n",
      "TODO"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Part 2: How to use pylearn2 to train an MLP\n",
      "\n",
      "Now that we've described the theory of what we're going to do, it's time to do it! This part describes\n",
      "how to use pylearn2 to run the algorithms described above.\n",
      "\n",
      "As in the MLP tutorial, we will use the convolutional net to do optical character recognition on the MNIST dataset.\n",
      "\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## pylearn2 Spaces\n",
      "\n",
      "In many places in pylearn2, we would like to be able to process several different kinds of data. In previous tutorials, we've just talked about data that could be preprocessed into a vector representation. Our algorithms all worked on vector spaces. However, it's often useful to format data in other ways. The pylearn2 Space object is used to specify the format for data. The VectorSpace class represents the typical vector formatted data we've used so far. The only thing it needs to encode about the data is its dimensionality, i.e., how many elements the vector has. In this tutorial we will start to explicitly represent images as having 2D structure, so we need to use the Conv2DSpace. The Conv2DSpace object describes how to represent a collection of images as a 4-tensor.\n",
      "\n",
      "One thing the Conv2DSpace object needs to describe is the shape of the space--how big is the image in terms of rows and columns of pixels? Also, the image may have multiple channels. In this example, we use a grayscale input image, so the input only has one channel. Color images require three channels to store the red, green, and blue pixels at each location. We can also think of the output of each convolution layer as living in a Conv2DSpace, where each kernel outputs a different channel. Finally, the Conv2DSpace specifies what each axis of the 4-tensor means. The default is for the first axis to index over different examples, the second axis to index over channels, and the last two to index over rows and columns, respectively. This is the format that theano's 2D convolution code uses, but other libraries exist that use other formats and we often need to convert between them."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## MNIST classification example"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Setting up a convolutional network in pylearn2 is essentially the same as setting up any other MLP. In the YAML experiment description below, there are really just two things to take note of.\n",
      "\n",
      "First, rather than using \"nvis\" to specify the input that the MLP will take, we use a parameter called \"input_space\". \"nvis\" is actually shorthand; if you pass an integer n to nvis, it will set input_space to VectorSpace(n). Now that we are using a convolutional network, we need the input to be formatted as a collection of images so that the convolution operator will have a 2D space to work on.\n",
      "\n",
      "Second, we make a few layers of the network be \"ConvRectifiedLinear\" layers. Putting some convolutional layers in the network makes those layers invariant to small translations, so the job of the remaining layers is much easier.\n",
      "\n",
      "We don't need to do anything special to make the Softmax layer on top work with these convolutional layers. The MLP class will tell the Softmax class that its input is now coming from a Conv2DSpace. The Softmax layer will then use the Conv2DSpace's convert method to convert the 2D output from the convolutional layer into a batch of vector-valued examples.\n",
      "\n",
      "The model and training is defined in conv.yaml file. Here we load it and set some of it's hypyer-parameters."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train = open('conv.yaml', 'r').read()\n",
      "train_params = {'train_stop': 50000,\n",
      "                    'valid_stop': 60000,\n",
      "                    'test_stop': 10000,\n",
      "                    'batch_size': 100,\n",
      "                    'output_channels_h2': 64, \n",
      "                    'output_channels_h3': 64,  \n",
      "                    'max_epochs': 500,\n",
      "                    'save_path': '.'}\n",
      "train = train % (train_params)\n",
      "print train"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "!obj:pylearn2.train.Train {\n",
        "    dataset: &train !obj:pylearn2.datasets.mnist.MNIST {\n",
        "        which_set: 'train',\n",
        "        start: 0,\n",
        "        stop: 50000\n",
        "    },\n",
        "    model: !obj:pylearn2.models.mlp.MLP {\n",
        "        batch_size: 100,\n",
        "        input_space: !obj:pylearn2.space.Conv2DSpace {\n",
        "            shape: [28, 28],\n",
        "            num_channels: 1\n",
        "        },\n",
        "        layers: [ !obj:pylearn2.models.mlp.ConvRectifiedLinear {\n",
        "                     layer_name: 'h2',\n",
        "                     output_channels: 64,\n",
        "                     irange: .05,\n",
        "                     kernel_shape: [5, 5],\n",
        "                     pool_shape: [4, 4],\n",
        "                     pool_stride: [2, 2],\n",
        "                     max_kernel_norm: 1.9365\n",
        "                 }, !obj:pylearn2.models.mlp.ConvRectifiedLinear {\n",
        "                     layer_name: 'h3',\n",
        "                     output_channels: 64,\n",
        "                     irange: .05,\n",
        "                     kernel_shape: [5, 5],\n",
        "                     pool_shape: [4, 4],\n",
        "                     pool_stride: [2, 2],\n",
        "                     max_kernel_norm: 1.9365\n",
        "                 }, !obj:pylearn2.models.mlp.Softmax {\n",
        "                     max_col_norm: 1.9365,\n",
        "                     layer_name: 'y',\n",
        "                     n_classes: 10,\n",
        "                     istdev: .05\n",
        "                 }\n",
        "                ],\n",
        "    },\n",
        "    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {\n",
        "        batch_size: 100,\n",
        "        learning_rate: .01,\n",
        "        learning_rule: !obj:pylearn2.training_algorithms.learning_rule.Momentum {\n",
        "            init_momentum: 0.5,\n",
        "        },\n",
        "        monitoring_dataset:\n",
        "            {\n",
        "                'valid' : !obj:pylearn2.datasets.mnist.MNIST {\n",
        "                              which_set: 'train',\n",
        "                              start: 50000,\n",
        "                              stop:  60000\n",
        "                          },\n",
        "                'test'  : !obj:pylearn2.datasets.mnist.MNIST {\n",
        "                              which_set: 'test',\n",
        "                              stop: 10000\n",
        "                          }\n",
        "            },\n",
        "        cost: !obj:pylearn2.costs.cost.SumOfCosts { costs: [\n",
        "            !obj:pylearn2.costs.cost.MethodCost {\n",
        "                method: 'cost_from_X'\n",
        "            }, !obj:pylearn2.costs.mlp.WeightDecay {\n",
        "                coeffs: [ .00005, .00005, .00005 ]\n",
        "            }\n",
        "            ]\n",
        "        },\n",
        "        termination_criterion: !obj:pylearn2.termination_criteria.And {\n",
        "            criteria: [\n",
        "                !obj:pylearn2.termination_criteria.MonitorBased {\n",
        "                    channel_name: \"valid_y_misclass\",\n",
        "                    prop_decrease: 0.50,\n",
        "                    N: 10\n",
        "                },\n",
        "                !obj:pylearn2.termination_criteria.EpochCounter {\n",
        "                    max_epochs: 500\n",
        "                },\n",
        "            ]\n",
        "        },\n",
        "    },\n",
        "    extensions:\n",
        "        [ !obj:pylearn2.train_extensions.best_params.MonitorBasedSaveBest {\n",
        "             channel_name: 'valid_y_misclass',\n",
        "             save_path: \"./convolutional_network_best.pkl\"\n",
        "        }, !obj:pylearn2.training_algorithms.learning_rule.MomentumAdjustor {\n",
        "            start: 1,\n",
        "            saturate: 10,\n",
        "            final_momentum: .99\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "\n",
        "\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we use pylearn2's yaml_parse.load to construct the Train object, and run its main loop. The same thing could be accomplished by running pylearn2's train.py script on a file containing the yaml string.\n",
      "\n",
      "Execute the next cell to train the model. This will take several minutes and possible as much as a few hours depending on how fast your computer is."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pylearn2.config import yaml_parse\n",
      "train = yaml_parse.load(train)\n",
      "train.main_loop()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Parameter and initial learning rate summary:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "Using gpu device 0: GeForce GTX 285\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tb: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tW: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tb: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tsoftmax_b: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tsoftmax_W: 0.00999999977648\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling sgd_update done. Time elapsed: 28.596286 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "compiling begin_record_entry done. Time elapsed: 0.118163 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitored channels: \n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum...\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "graph size: 140\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "graph size: 138\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Compiling accum done. Time elapsed: 25.629903 seconds\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 0\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.499999672174\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.166490137577\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.141212999821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.121112905443\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.18473732471\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.15763354301\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.12533462048\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 2.31326198578\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 2.3082048893\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00505703082308\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.147996068001\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.134659647942\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.119130179286\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.897199928761\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.166490137577\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.141212999821\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.121112905443\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.18473732471\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.15763354301\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.12533462048\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 2.3118827343\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 2.30682563782\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00505703082308\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.147762432694\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.134393379092\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.118998683989\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.89100009203\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.497043\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 1\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 50000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.499999672174\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.545692265034\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.299479395151\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.130041897297\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.25489199162\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.1928191185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13390684128\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.205174013972\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.199306234717\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00586774758995\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999898374081\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.907166719437\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.375571638346\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.059200014919\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.545692265034\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.299479395151\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.130041897297\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.25489199162\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.1928191185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13390684128\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.203009739518\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.197141975164\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00586774758995\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999921679497\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.910249471664\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.369747459888\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0575000010431\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.463996\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 2\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.554444551468\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.649543344975\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.332886070013\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.133387058973\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.27290606499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.20431303978\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13722503185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0992075428367\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0930676087737\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00613996293396\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999974370003\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.956771910191\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.501736760139\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0267999880016\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.649543344975\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.332886070013\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.133387058973\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.27290606499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.20431303978\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13722503185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.10961329937\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.103473365307\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00613996293396\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999980568886\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.95684325695\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.460102945566\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0298999808729\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.424147\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 3\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 1500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.608888924122\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.704413831234\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.352466374636\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.134819567204\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.29450547695\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.21162760258\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13987505436\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0756950974464\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0693778172135\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00631728721783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999986827374\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.968378245831\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.528870224953\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0213999953121\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.704413831234\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.352466374636\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.134819567204\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.29450547695\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.21162760258\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13987505436\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0831698328257\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0768525525928\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00631728721783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999992907047\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.967627465725\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.484717309475\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.022599991411\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.431855\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 4\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.663333714008\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.750690817833\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.367417871952\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.136356577277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.31281983852\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.21794211864\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.14188706875\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.068290501833\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0618221834302\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00646831514314\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999996840954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.974141418934\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.556996643543\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0203999951482\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.750690817833\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.367417871952\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.136356577277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.31281983852\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.21794211864\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.14188706875\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0721263736486\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0656580626965\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00646831514314\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997437\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.973989069462\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.518289446831\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0197999943048\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.846816\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 5\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 2500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.717777192593\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.78807425499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.380661904812\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.138193532825\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.32829785347\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.22380626202\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.14132940769\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0541290827096\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0475195422769\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00660955067724\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999997019768\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.978952407837\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.586321294308\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0147999990731\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.78807425499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.380661904812\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.138193532825\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.32829785347\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.22380626202\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.14132940769\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0632437169552\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0566341690719\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00660955067724\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999997496605\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.978959202766\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.543452858925\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0152999954298\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.220607\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 6\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.772221684456\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.83024930954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.393128633499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.14109762013\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.34349620342\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.23068916798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13882601261\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.046277385205\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0395119972527\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00676536094397\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999998569489\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.983581066132\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.61893260479\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0118999946862\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.83024930954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.393128633499\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.14109762013\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.34349620342\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.23068916798\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13882601261\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0579625852406\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0511972345412\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00676536094397\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999998748302\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.982799112797\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.570958197117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.014899995178\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.435793\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 7\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 3500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.826667308807\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.874054849148\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.404089450836\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.145140781999\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.3592710495\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.23862838745\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13550460339\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0618967972696\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0549634955823\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00693332310766\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.979612290859\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.568928360939\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.018599992618\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.874054849148\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.404089450836\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.145140781999\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.3592710495\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.23862838745\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13550460339\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0732903108001\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0663569867611\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00693332310766\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.99999922514\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.980143547058\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.554540336132\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0205999948084\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.197597\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 8\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.881111502647\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.918360114098\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.41432377696\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.152310580015\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.38522279263\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.24944329262\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.13081908226\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0419789850712\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0348266772926\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00715232128277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.99999922514\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.984818339348\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.647026181221\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.010899996385\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.918360114098\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.41432377696\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.152310580015\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.38522279263\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.24944329262\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.13081908226\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0546835511923\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0475312396884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00715232128277\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.99999922514\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.984225571156\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.609494626522\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0131999971345\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.257716\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 9\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 4500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.935554862022\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 0.983334720135\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.426948130131\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.163345575333\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.44658946991\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.27363669872\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.12254095078\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0422964543104\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0347021482885\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.00759429484606\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999284744\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.987657427788\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.674763560295\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0109999952838\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 0.983334720135\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.426948130131\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.163345575333\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.44658946991\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.27363669872\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.12254095078\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0550039000809\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.047409594059\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.00759429484606\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999284744\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.987751126289\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.627986073494\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0135999955237\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.346043\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 10\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.44428324699\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.638218581676\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.226101309061\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.62931704521\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.08986961842\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0722205787897\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.060090906918\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0121296364814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.982179641724\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.605987489223\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.018699998036\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.44428324699\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.638218581676\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.226101309061\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.62931704521\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.08986961842\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0778582692146\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0657286569476\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0121296364814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.982598662376\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.571307122707\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.018899993971\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.230181\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 11\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 5500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 550000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.67923891544\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.759219646454\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.222079560161\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.68946993351\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 1.03812360764\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0509216226637\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0374662131071\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.013455402106\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.987638652325\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.641136288643\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0125999962911\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.67923891544\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.759219646454\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.222079560161\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.68946993351\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 1.03812360764\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0591181144118\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.045662689954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.013455402106\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.987159013748\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.612255334854\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0139999939129\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.166241\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 12\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 6000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 600000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.79883503914\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.791452467442\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.214680939913\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.69265186787\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.990326702595\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0495410040021\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0358138717711\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0137271359563\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.990619242191\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.677489697933\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0121999960393\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.79883503914\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.791452467442\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.214680939913\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.69265186787\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.990326702595\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0574188157916\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0436916723847\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0137271359563\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.991107225418\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.666162133217\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0119999954477\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.133832\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 13\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 6500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 650000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.83014798164\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.801658511162\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.207240864635\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.68411743641\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.943459570408\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0586318746209\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0448817498982\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0137501331046\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.986112833023\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.627143979073\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0142999952659\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.83014798164\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.801658511162\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.207240864635\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.68411743641\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.943459570408\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0668870285153\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0531368739903\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0137501331046\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.986647307873\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.608526527882\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0150999957696\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.391329\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 14\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 7000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 700000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.87506687641\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.830554485321\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.204696804285\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.67928016186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.898255288601\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0461465120316\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0322162732482\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0139302331954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.991206288338\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.680966556072\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00969999749213\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.87506687641\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.830554485321\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.204696804285\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.67928016186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.898255288601\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0573299974203\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0433997660875\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0139302331954\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.991953551769\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.683482408524\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0114999953657\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.137590\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 15\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 7500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 750000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.91566932201\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.83042216301\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.199055761099\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.66764605045\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.854915261269\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0496817082167\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0358328558505\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0138488588855\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.990178763866\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.676120936871\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0115999961272\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.91566932201\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.83042216301\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.199055761099\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.66764605045\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.854915261269\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0512396655977\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0373908095062\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0138488588855\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.990789115429\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.658932745457\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0109999962151\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.368612\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 16\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 8000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 800000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.838400304317\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.194422900677\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.65690112114\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.813605189323\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0543208941817\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0404779799283\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0138429058716\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.987714588642\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.635551512241\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.012999993749\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.9364978075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.838400304317\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.194422900677\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.65690112114\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.813605189323\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0548343248665\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0409914143384\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0138429058716\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.989114284515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.642774403095\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0112999966368\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.209649\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 17\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 8500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 850000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.91171455383\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.846678137779\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.19338837266\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.64647006989\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.774317622185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.046179626137\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0323496200144\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0138300256804\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.991029918194\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.694849073887\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0107999956235\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.91171455383\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.846678137779\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.19338837266\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.64647006989\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.774317622185\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0499480105937\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0361180007458\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0138300256804\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.991879045963\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.68752592802\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00959999952465\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.658055\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 18\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 9000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 900000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.850539088249\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.189176186919\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.6371011734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.737512171268\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0443185269833\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.030502744019\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0138157960027\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.992875516415\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.736675024033\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00939999707043\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.850539088249\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.189176186919\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.6371011734\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.737512171268\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.053897023201\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0400812476873\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0138157960027\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.99310708046\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.703199386597\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0110999941826\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.333906\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 19\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 9500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 950000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.847409486771\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.186600789428\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.61723184586\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.701808393002\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0403720587492\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0267571881413\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.013614885509\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.993013560772\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.736813545227\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00850000046194\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.847409486771\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.186600789428\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.61723184586\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.701808393002\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0431119017303\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0294970460236\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.013614885509\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.993714511395\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.717002928257\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0089999968186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.340939\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 20\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 10000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1000000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.92669391632\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.845552861691\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.181986898184\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.6023042202\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.668366849422\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0428223386407\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0293126199394\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0135097391903\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.993188142776\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.739611268044\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0105999978259\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.92669391632\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.845552861691\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.181986898184\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.6023042202\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.668366849422\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.050708103925\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0371983982623\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0135097391903\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.99406504631\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.734104990959\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00999999605119\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.977390\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 21\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 10500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1050000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.93582856655\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.848804593086\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.178966462612\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.59039211273\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.635490596294\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0458019226789\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0323437191546\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0134581970051\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.991329133511\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.691799819469\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0105999968946\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.93582856655\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.848804593086\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.178966462612\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.59039211273\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.635490596294\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0614379122853\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0479796975851\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0134581970051\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.993205845356\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.710878610611\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0122999958694\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.437323\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 22\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 11000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1100000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.90647602081\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.848101198673\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.17422285676\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.58007156849\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.604269385338\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0386426523328\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0252483561635\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0133942868561\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.993958890438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.751003742218\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00759999686852\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.90647602081\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.848101198673\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.17422285676\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.58007156849\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.604269385338\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.048912987113\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0355186872184\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0133942868561\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.994857549667\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.738139390945\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0094999987632\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:44.925952\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 23\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 11500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1150000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.91347312927\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.844412982464\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.166331276298\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93618237972\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.55887579918\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.574603140354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0355288870633\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0223389472812\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0131899472326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.994695007801\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.767133533955\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00739999813959\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.91347312927\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.844412982464\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.166331276298\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93618237972\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.55887579918\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.574603140354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0428306758404\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0296407323331\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0131899472326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995229959488\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.754100978374\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00719999661669\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.135969\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 24\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 12000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1200000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.91951978207\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.836171507835\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.164778992534\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.53445827961\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.546444058418\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0356558673084\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.022733990103\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0129218865186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.994695842266\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.762125074863\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00759999779984\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.91951978207\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.836171507835\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.164778992534\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.53445827961\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.546444058418\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0455874875188\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0326656289399\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0129218865186\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.994600176811\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.740817070007\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00879999715835\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.088647\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 25\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 12500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1250000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.93276119232\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.831019997597\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.161167949438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.52135765553\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.519681036472\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0410521477461\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0282514858991\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0128006627783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.994140505791\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.761759340763\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0077999974601\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.93276119232\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.831019997597\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.161167949438\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.52135765553\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.519681036472\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0435176976025\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0307170264423\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0128006627783\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.994168877602\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.740992307663\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00889999885112\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.153518\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 26\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 13000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1300000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.92989122868\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.825959026814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.156162112951\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93493390083\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.50425136089\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.494196981192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.036505330354\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0238700080663\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0126353390515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.995713174343\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.775741994381\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00739999813959\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.92989122868\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.825959026814\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.156162112951\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93493390083\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.50425136089\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.494196981192\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0445033796132\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0318680629134\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0126353390515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995958864689\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.788133025169\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00759999779984\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.499386\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 27\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 13500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1350000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.92615044117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.837450802326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.148752957582\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.50518739223\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.470022141933\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0425866432488\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0297948271036\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0127918105572\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.99336540699\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.709817945957\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00999999698251\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.92615044117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.837450802326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.148752957582\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.50518739223\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.470022141933\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0502808466554\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0374890305102\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0127918105572\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995107114315\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.763227462769\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00939999707043\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.068647\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 28\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 14000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1400000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.89880204201\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.835507690907\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.141381591558\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.49202251434\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.447112351656\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0373257026076\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0246458798647\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0126798404381\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.994658648968\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.7678809762\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00809999834746\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.89880204201\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.835507690907\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.141381591558\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.49202251434\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.447112351656\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0448061972857\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0321263670921\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0126798404381\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.994764626026\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.737609565258\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00789999775589\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:44.940250\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 29\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 14500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1450000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.89456033707\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.826852142811\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.1342484653\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.47248113155\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.425320059061\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0488765016198\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0364154167473\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.012461095117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.991527915001\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.708343684673\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.0109999962151\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.89456033707\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.826852142811\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.1342484653\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.47248113155\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.425320059061\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0598892904818\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0474282093346\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.012461095117\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.992517650127\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.702422738075\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.0121999951079\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.033842\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 30\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 15000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1500000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.89785075188\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.833137273788\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.12766776979\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.46606051922\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.404478013515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0398592054844\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0273652821779\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0124939084053\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.99489235878\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.767376661301\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00859999842942\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.89785075188\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.833137273788\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.12766776979\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93649816513\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.46606051922\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.404478013515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0449932217598\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0324993059039\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0124939084053\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995154082775\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.750459074974\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00799999665469\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.072603\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 31\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 15500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1550000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.88594102859\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.822186350822\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.121126919985\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.93575370312\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.44667637348\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.384828656912\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0402425862849\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0279870275408\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0122555522248\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.993545353413\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.723762392998\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00869999732822\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.88594102859\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.822186350822\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.121126919985\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.93575370312\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.44667637348\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.384828656912\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0425297468901\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0302741993219\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0122555522248\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.994219899178\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.708793342113\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00829999800771\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:45.377313\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 32\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 16000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1600000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.87813127041\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.814654231071\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.11509488523\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.92595422268\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.42374145985\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.366239726543\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0316884964705\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0196874346584\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0120010524988\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.995534360409\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.770359456539\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00539999874309\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.87813127041\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.814654231071\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.11509488523\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.92595422268\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.42374145985\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.366239726543\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0424967259169\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0304956696928\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0120010524988\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995728135109\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.783714473248\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00799999758601\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Time this epoch: 0:01:44.896808\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Monitoring step:\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tEpochs seen: 33\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tBatches seen: 16500\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tExamples seen: 1650000\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tlearning_rate: 0.00999999046326\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tmomentum: 0.989998817444\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_max: 1.87118041515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_mean: 0.804925084114\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h2_kernel_norms_min: 0.109122686088\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_max: 1.90657293797\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_mean: 1.3938267231\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_h3_kernel_norms_min: 0.348352521658\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_objective: 0.0377874299884\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_0: 0.0261260885745\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_term_1_weight_decay: 0.0116613674909\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_mean_max_class: 0.99490904808\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_min_max_class: 0.76411986351\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\ttest_y_misclass: 0.00859999842942\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_max: 1.87118041515\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_mean: 0.804925084114\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h2_kernel_norms_min: 0.109122686088\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_max: 1.90657293797\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_mean: 1.3938267231\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_h3_kernel_norms_min: 0.348352521658\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_objective: 0.0444900207222\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_0: 0.0328286737204\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_term_1_weight_decay: 0.0116613674909\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_max_max_class: 0.999999344349\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_mean_max_class: 0.995154678822\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_min_max_class: 0.737455904484\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\tvalid_y_misclass: 0.00819999724627\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Compiling the theano functions used to run the network will take a long time for this example. This is because the number of theano variables and ops used to specify the computation is relatively large. There is no single theano op for doing max pooling with overlapping pooling windows, so pylearn2 builds a large expression graph using indexing operations to accomplish the max pooling."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "After the model is trained, we can use the print_monitor script to print the last monitoring entry of a saved model. By running it on \"convolutional_network_best.pkl\", we can see the performance of the model at the point where it did the best on the validation set."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!print_monitor.py convolutional_network_best.pkl | grep test_y_misclass"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": "*"
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The test set error has dropped to 0.74%! This is a big improvement over the standard MLP.\n",
      "\n",
      "We can also look at the convolution kernels learned by the first layer, to see that the network is looking for shifted versions of small pieces of penstrokes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!show_weights.py convolutional_network_best.pkl"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Further reading\n",
      "\n",
      "You can find more information on convolutional networks from the following sources:\n",
      "\n",
      "[LISA lab's Deep Learning Tutorials: Convolutional Neural Networks (LeNet)](http://deeplearning.net/tutorial/lenet.html)\n",
      "\n",
      "\n",
      "This is by no means a complete list."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}
